Give the census guard test teeth, and stop it corrupting the census - #100
Conversation
#96's fix for the vacuous guard test moved the defect rather than removing it. `PrefixListTests.setUp` still imported prefix_census in-process, and that is the import under test: with the guard removed the scan ran in setUp, clobbering the census *before* the test read `before`, after which the subprocess re-ran the same deterministic scan and produced byte-identical output. The assertion compared a corrupted file against itself and passed (#98). My mutation test missed this because it used the real corpus, where the scan exceeds the 60-second timeout. Against a ten-file corpus the old test reports OK in 0.057s while the census md5 changes underneath it. Only the timeout ever had teeth, and only because the corpus happens to be slow. Worse, the child computes its paths from `__file__`, so a regression would have an unguarded interpreter write the real tracked census and then take SIGKILL at sixty seconds, possibly mid-`json.dump`. The test written to prevent #95 could cause it. The guard test now lives in its own TestCase with no setUp importing the module, so the snapshot is genuinely taken first, and the child runs with MECHS_ROOT pointing at an empty directory. An unguarded import then reaches roots.mech_root, which exits rather than counting an empty corpus, so the child fails in milliseconds: deterministic, independent of machine speed, and nothing writes to a tracked file. Against the ten-file corpus that defeated the old version it now fails in 0.048s with the census untouched. stderr is captured rather than discarded, so a failure reports the traceback. `literal()` also only saw `ast.Assign`, so four ways to rebind a list after its literal survived it (#99) — `VOC += [...]`, `VOC.append(...)`, `VOC[0] = ...` and a rebinding nested in an `if`. It now walks the tree for Assign, AugAssign and AnnAssign, and separately fails on subscript assignment or a mutating method call. All five shapes are killed; the commit message in 39122af claiming "a later reassignment cannot hide" was true only of the plain `=` form. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Splitting the guard test into its own class fixed its detection but closed only half of #98. `PrefixListTests.setUp` still did an in-process `import prefix_census`, so against a regressed module the suite went on overwriting the tracked census on every run — the guard test would now fail, correctly, but the damage had already happened in setUp. Measured on a copy with the guard removed and a ten-file corpus: the suite reported the failure and the census md5 changed anyway. setUp now fetches `P` and `norm` from a separate interpreter, the same way the guard test checks the import, with `MECHS_ROOT` pointing at an empty directory. A regressed module dies there in milliseconds and setUp raises with the child's stderr, so the failure is loud and nothing is written. Same corpus, same mutation, now: five failures and errors, and the census byte-identical. Reading the constants out of a subprocess rather than parsing them from source keeps the values the pipeline actually uses, which is what makes the accepted set exactly the 53 keys the census can emit rather than an approximation of it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The child derived its own REPO from `__file__`, so its DATA was the real
tracked `_fleet/data`. Nothing was written there only because
json.dump(census(), open(DATA + "/prefix_census.json", "w"), indent=1)
evaluates `census()` — which exits under an empty MECHS_ROOT — before `open()`
truncates. That is an accident of argument order, and the expression emits a
ResourceWarning on every real run, which actively invites the idiomatic
with open(DATA + "/prefix_census.json", "w") as fh:
that reverses it. With both that refactor and the guard regression applied, the
suite's own children truncated the tracked census to zero bytes.
The child now runs against a copy of scripts/fleet in a temp tree with its own
_fleet/data, so REPO and DATA resolve inside the sandbox whatever the module
does. Same mutation, now: four failures and an error, and the tracked census
byte-identical at 2,778.
That also allows a better assertion. Comparing the real file's bytes is vacuous
once the child cannot reach it; the guard now lists what the child wrote in the
sandbox and requires it to be empty — positive evidence, and independent of the
exit code. The byte comparison stays as a backstop.
`PrefixListTests.setUp` shares the same sandbox rather than duplicating the
launch.
Also: a comment claimed the child's failure surfaced through `check=True`,
which this hunk had deliberately removed in favour of asserting on returncode
so the child's stderr reaches the failure message. In a test whose whole value
is explaining why it has teeth, that was the sentence a reader would act on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adversarial reviewTwo independent lenses — the guard mechanism, and Confirmed and fixed1. The split closed detection but left the corruption path open (
Worse under pytest, which orders classes differently:
2. The child could truncate the tracked census, and was safe only by accident ( The child derives json.dump(census(), open(DATA + "/prefix_census.json", "w"), indent=1)evaluates Measured with that refactor plus the guard regression: the suite's own children truncated the tracked census to 0 bytes. The child now runs against a copy of That also bought a better assertion: comparing the real file's bytes is vacuous once the child is sandboxed, so the guard now lists what the child wrote in the sandbox and requires it to be empty — positive evidence, independent of the exit code. Filed as #102, since the underlying line is unchanged and reachable outside tests. 3. The call check fired on any attribute call, so
The "bound N times" message now names the offending lines, so a shadowing local is obvious rather than baffling. Filed, not fixed
Corrections to my own record
Negative resultsNo way found to make an unguarded import exit 0; no false-failure path for a guarded one ( One residual, judged negligible: 27 tests, |
1 participant
Closes #98 and #99. Follow-up to #96, which merged before its review returned.
The guard test could still not fail
#96's fix moved the defect rather than removing it.
PrefixListTests.setUpstill importedprefix_censusin-process — and that import is the thing under test. With the guard removed:setUp's import runs the scan and clobbers the censusbefore— already the clobbered bytesMy mutation test in #96 missed this because it used the real corpus, where the scan exceeds the 60-second timeout. Only the timeout ever had teeth — and only because the corpus happens to be slow.
And it could corrupt the repository. The child computes its paths from
__file__, so a regression would have an unguarded interpreter write the real tracked census and then take SIGKILL at sixty seconds, possibly mid-json.dump. The test written to prevent #95 could cause it.Fix
The guard test now lives in its own TestCase with no
setUpimporting the module, so the snapshot is genuinely taken first — and the child runs withMECHS_ROOTpointing at an empty directory. An unguarded import then reachesroots.mech_root, which exits rather than counting an empty corpus, so the child fails in milliseconds: deterministic, machine-speed independent, no timeout involved, nothing written to a tracked file.Against the ten-file corpus that defeated the old version:
stderris captured rather than discarded, so a failure reports the traceback instead of only "non-zero exit status 1".literal()missed four rebinding shapesIt collected module-level
ast.Assignonly, so these survived — the test kept reading the original list while the module used another:VOC = VOC + ["BOGUSVOC"]VOC += ["BOGUSVOC"]VOC.append("BOGUSVOC")VOC[0] = "BOGUSVOC"if True:It now walks the tree for
Assign,AugAssignandAnnAssign, and separately fails on subscript assignment or a mutating method call.39122af's claim that "a later reassignment cannot hide" was true only of the plain=form.Provenance
Found by an adversarial review agent that returned after #96 merged. I had verified the wrong thing — a slow corpus — which masked it. The real fix for the underlying awkwardness is #97: make
build_subsets.pyandbuild_data.pyimportable so the tests read actual objects instead of parsing text.27 tests pass,
assemble_page.py --checkclean.🤖 Generated with Claude Code